APPLYING ONLINE PEER 

ASSESSMENT WITH TOTAL 
QUALITY MANAGEMENT TO 
ELEVATE PROJECT-BASED 
LEARNING PERFORMANCE 


Shu-Hsuan Chang, 
Li-Chih Yu, 
Yen-Kuang Kuo, 
Yi-Ting Mai, 
Jen-De Chen 


Introduction 


In the education of undergraduate science, technology, engineering, 
and mathematics (STEM) in Taiwan, projects are an essential parts of the 
curriculum for training students in project-based learning as well as col- 
laborative/cooperative learning. This is particularly true to engineering 
related programs because engineering education plays an important role 
in developing students’ product design proficiency in response to demand 
of the industrial ear (Tseng, Chang, Lou & Chen, 2013). 

The requisite skills of an engineer include coping with technical com- 
plexities, evolving solutions, working successfully in the future (Pospisil, 
1996; Brown & Eisenhardt, 1995; Drucker, 1998; Sherma, 1999) by commu- 
nicating effectively and performing well in interdisciplinary teams (Hirst 
& Mann, 2004; Sawhney & Prandelli, 2000; Sen & Engelhoff, 2000), and 
generating and commercializing innovative ideas which satisfy customer 
needs (Tomkovich & Miller, 2000; Debruyne et al., 2002; Thamhain, 2003; 
Hirst & Mann, 2004). Therefore, educators and institutes have been devoted 
to develop integrated education programs and learning strategies for STEM 
to achieve the aforementioned goals. 

Project-Based Learning (PBL) enables the simulation of the real world 
phenomena, prepare students for future industrial practice. PBL is a learn- 
ing method which highlights “learning by doing”. Through the learning 
activities, learners take the initiative to identify problems, define questions, 
and find solutions (Liang, 2012). Learners can be equipped with knowledge 
and skills, as well as apply them to complete the project (Blumenfeld et al., 
1991; Barrows, 1996; Thomas et al., 1999). However, some studies indicated 
that it is critical for PBL to adopt mutual team cooperation to bring up 
effective learning (Krajcik et al., 1994). Project participants must interact 
with people inside and outside of the project to gain adequate informa- 
tion, plan for strategies and actions, and promote their learning efficiently 
through learning from each team and competing with other teams (Poell 
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Abstract. Undergraduate science, technol- 
ogy, engineering, and mathematics (STEM) 
curriculum emphasize project-based learn- 
ing (PBL) with peer assessment/on-line peer 
assessment (PA/OPA). Many studies have 
stressed that students did not improve over 
two rounds in PBL with OPA studies and 
PBL with PA have to adopt team mutual 
cooperation to reap effective learning. This 
study proposed an innovative approach 
that incorporate OPA with TQM as a macro- 
level instructional tool to guide students in 
teacher's directing of collaborative project 
development as well as seeking continu- 
ous improvement to elevate Project-Based 
Learning Performance in a STEM course. 
The effects of OPA with TQM were examined 
through an experiment with PBL perform- 
ances hypotheses. A total of 63 junior stu- 
dents in an university of Taiwan voluntarily 
participated in this study and a quasi- 
experimental approach with a two-group 
design was adopted. The results revealed 
that the team members using the OPA 

with TQM approach tended to have higher 
design skill performance, better cohesive 
teamwork and creative problem solving 
attitude. Thus, the proposed approach 
facilitated team members to collaborate for 
seeking continuous improvement. However, 
no significant difference was reported on 
the influence of enhancing students’ design 
concept. Implication and suggestions for 
educators to promote the PBL with OPA and 
TQM were also provided in the study. 
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& Van der Krogt, 2003). However, teamwork problems such as insufficient communication, uneven distribution 
of tasks, and the free-rider problems are also significant issues in student projects that jeopardized teamwork 
quality (Hoegl & Gemuenden, 2001). 

The meanings of collaborative learning and cooperative learning are different. Brufee (1999) advocated that 
collaborative learning and cooperative learning have same goal to assist students while learning together in the 
journey of important knowledge and skills acquisition. The most vital factor of these two learning approaches 
is the facilitation of constructive communication among team members to achieve the learning objectives that 
a single person could not afford. According to Merriam-Webster’s New International Dictionary, cooperation is 
defined as people work together through labor division and coordination. For example, if a group of authors 
wants to write a book, cooperation means that each member in the group who has his/her own accountability 
(e.g., one chapter) writes his/her accountable chapter by himself/herself rather than completing the whole 
book. Meanwhile, collaboration emphasizes common contribution instead of working together on the division 
of work. Therefore, if a group of authors wants to write a book collaboratively, they would write each chapter 
altogether. 

Cooperative learning is an educational method that students of mixed levels of ability are arranged into 
groups and will be rewarded according to the aggregative success of the group rather than that of each individual 
member. Collaborative learning is also an educational treatment that more than two students work together to 
learn something. It is based on a general premise that groups of students can learn better from each other through 
sharing and social interaction than they would if they learned on their own. The collaborative learning concept 
is based on the idea that diversity of knowledge and experience positively impacts the learning outcomes. 

Springer, Stanne and Donovan (1999) poised that “collaborative learning is characterized by relatively un- 
structured processes through which participants negotiate goals, define problems, develop procedures, and produce 
socially constructed knowledge in small groups (p. 24)”. Moreover, according to the literatures of ERIC, the subjects 
of collaborative learning are usually undergraduate and graduate students, because collaborative learning aims 
to flip the traditional classrooms up especially in higher education settings where professors have supreme 
authority in that social structure. This traditional social structure might diminish students’ freedom of thoughts 
and obstruct their possibility of creativity. The adoption of collaborative learning could provide students with 
autonomy, encourage them to speak, change their passive learning attitude into active ones, and enhance 
students’ social skills, and cultivate the maturity of wisdom. Thus, the work reported in this paper focused on 
the collaborative learning approach. However, teachers applying the collaborative learning techniques in PBL, 
whether in the traditional or online forms, all encountered many issues. Challenges in collaborative learning such 
as insufficient communication, uneven distribution of tasks, dominance of a single member in a group (Williams 
& Karau, 1991), free-rider predicament (Strong & Anderson, 1990), and the sucker effect (Steiner, 1972) are all 
significant challenges in student projects, impacting the effective of teamwork as well as individual learning 
performance. The design of teamwork training and team management for capstone PBL must be structured to 
prevent the aforementioned problematic issues from arising. 

Johnson and Johnson (1987) stressed that successful cooperative learning consists of five elements: (1) posi- 
tive interdependent relationship among team members; (2) face-to-face interaction; (3) individual accountability; 
(4) collaborative skills training in advance; (5) monitoring and reflecting of group process. Hoegl and Gemuenden 
(2001) developed a comprehensive concept of collaboration in teams, termed Teamwork Quality (TWQ), which is 
conceptualized as having six dimensions: communication, coordination, balanced member contributions, mutual 
support, effort, and cohesion. Many researchers have attempted to respond to the aforementioned issues in the 
collaborative learning domain and Peer Assessment (PA) is a popular exemplary solution. 

Peer assessment refers to any arrangement in which learners consider the amount, level, value, worth, 
quality or success of the learning outcomes of their peers (Topping, 1998). With the rapid development of 
information technology, Online Peer Assessment (OPA) enabled students to operate reflective thinking and 
clarify their misconceptions by providing a comfortable learning environment that is free from temporal and 
spatial constraints, and that allows participants to work and be graded anonymously (McConnell, 1999). Zhao 
(2000) pointed out that although anonymity had made it possible for students to dare to speak and enhanced 
the usefulness of feedback, yet, anonymity is like double-edged sword that students would not work as hard as 
they would in their homework without anonymity. Many approaches were designed for the establishment of 
personal accountability for each team member and relations of positive interdependence among team members 
by resource and/or reward interdependence (Kao, Lin & Sun, 2008; Johnson & Johnson, 1992, 2009; Lin, Sun & 
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Kao, 2002; Kao, 2013; Chen, Hong, & Chen, 2014). Nevertheless, some students found frustrations in PA owing to 
lack of clear instruction plan, evaluation guideline, and good teamwork (Orsmond & Merry, 1996; Clifford, 1999). 
Moreover, since persistence for pursuing improvement is vital to the success of PBL with OPA (Lin et al., 2001; 
Tsai et al., 2002; Tseng & Tsai 2007). Lin et al. (2001) found that many students did not improve over two rounds 
in an OPA study. Both Tsai et al. (2002) and Tseng & Tsai (2007) showed a similar finding--most students improved 
their work over three-round, which was an optimal situation. Still, how to inspire students’ teamwork incentive, 
how to create their positive independence to fully participate in PBL with OPA, and how to encourage them to 
develop a continuous improvement processes are issues and threats for the performance of PBL. Capstone PBL 
for STEM education are supposed to help students to acquire knowledge and skills in how to function effectively 
in teams and to cultivate students’ spirits of continuous seeking improvement before entering the workforce. 

Total Quality Management (TQM) stresses improving product quality to meet customer needs by creating 
organization excellence at every level, through collaboration among team members, full participation, and con- 
tinuous improving processes. TOM is a practice of continuous improvement through the repetition of Deming 
Cycle to detect and solve problems (Edward, 1996; Chizmar, 1994). The core characteristics of TQM include: (1) 
customer-orientation (Bensimon, 1995; Ewell, 1993; Kanji & Asher, 1995; Tsang & Antony, 2001), (2) promoting 
teamwork by a Quality Control Circle (Bonstingl, 1993; Chizmar, 1994; Kanji & Asher, 1995; Tsang & Antony, 2001), 
(3) continuous improvement by the “Plan-Do-Study-Action” (PDSA) (Deming, 1986; Kanji & Asher, 1995; Tsang & 
Antony, 2001), and (4) correctly reflecting upon the efficiency of progress. 

TQM has been identified as an effective approach to enhance teamwork performance in education. Chizmar 
(1994) asserted that teaching strategy, integrating TQM, could elevate students’ active learning attitude. Students 
should personally participate in projects, and accumulate experiences in the course. Dahlgaard, Kristensen and 
Kanji (1995) indicated that an educational culture with TOM is created by increasing customer satisfaction through 
continuous improvements, in which all staffs and students actively participate. TQM may promote PBL objectives 
achieving which includes the precut of learners’ continuous self-evaluation and regular reflections upon whether 
or not to adjust the learning strategies (Blumenfeld et al., 1991). This loop of learning feedback will further foster 
the effectiveness of PBL groups collaborative learning (Krajcik et al., 1994; Lou & MacGregor, 2004). 

According to the aforementioned studies, OPA supported by TQM is a beneficial tool which might not only 
cultivate students’ spirits of continuous seeking improvement but also enhance their team work skills. TQM was 
grafted and integrated into OPA in the current study. this notion is also suitable for use with traditional peer 
assessment. The key assumption is that introducing TQM teamwork training and team management into the 
assessment process for capstone PBL may elevate learning performance. 

To further inspire students’ TOM spirit, the study introduced “continuous seeking improvement” into the 
OPA activity. This is done through the scoring formula used to assign grades. The principle involved can be il- 
lustrated with the following general formula. 


Total Score= (Final Score from Teacher)x(W1) + (Average Progressive Score of team project from all three-rounds of 
OPA)x(W2) 


Where Total Score=the final course grades; Final Score from Teacher = the final (the 37 round) score that each 
group receives from their teacher on the assignment; this score was based on the ratings of all participating 
peer assessors feedback; W1 = the proportional weight that the teacher allocated to the Final Score from Teacher, 
Average Progressive Score of our team project from all three-rounds of OPA = the summation of the progressive 
score from all three-rounds OPA and then was divided by three; W2 = the proportional weight that the teacher 
allocated to the Average Progressive Score of team project from all three-rounds of OPA. 

The formula comprised of two components: a “Final Score from Teacher" and an “Average Progressive Score 
of team project from all three-rounds of OPA."“Final Score from Teacher’ is a score which is determined by teacher 
after assessing the 3 round of group work for each group. The study included teacher's assessment as part of 
the total score because Wen & Tasi (2006) pointed out that most students preferred the PA scores to constitute 
a minor portion of the final score; none of the students favored total reliance merely on the PA score itself, and 
the responsibility of grading and summative evaluation should full upon the teacher’s shoulders. The second 
component, Average Progressive Score of team project from all three-rounds of OPA, encompassed an essential fac- 
tor, reward interdependence among team members, encouraging continuously improvement seeking. Moreover, 
reward interdependence prevented the reward system from being unidirectional (insiders only) and ensured 
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diverse sources of feedback; team members of different abilities and traits were all given a channel to provide 
feedback, in cooperative a multidimensional evaluation interface. 

Therefore, the purpose of the study was to introduce and examine the effectiveness of an innovative ap- 
proach for PBL, named “OPA with TQM,’ which was designed to combat the aforementioned threats of PBL and 
enhance students’ performance of PBL. The effectiveness of proposed OPA pedagogy were examined via an 
experiment. A total of 63 junior university students in their junior years in Taiwan voluntarily participated in this 
study and a two-group quasi-experimental approach, OPA with and without TQM treatment. The experiment 
design was a nonequivalent pretest-post-test control/comparison group approach with dependent variable 
being students’ performance in OLED design concept, OLED design skills, and the attitude toward PBL, were 
evaluated to demonstrate the effects of “OPA with TQM” approach. The study attempted to address the follow- 
ing research issues. 

1. The effectiveness of OPA with TQM approach in PBL upon students’ OLED design concept. 
2. The effectiveness of OPA with TQM approach in PBL upon students’ OLED design skills. 

3. The effectiveness of OPA with TQM approach in PBL upon students’ attitude toward PBL. 
4. Students’ perception about the effect of OPA with TQM approach in PBL. 

Based on the criteria for the design of constructivist learning environments (Honebein, 1996), the Construc- 
tivist Project-based Learning Environment Survey (CPLES) was developed to evaluate students’ attitude toward 
PBL, and its construct and psychometric measures have been empirically validated (Chang, 2006; Chang et al., 
2011). CPLES is conceptualized as having six facets: Inquiry learning, Reflective learning, Teamwork, Creative 
problem solving, Open-endedness, Authenticity. 

Reviewing the existing literature relevant to the research purpose as foundations of this study, the hypoth- 
eses were elaborated as the following. 

H1: OPA with TQM has a positive effect on students’ design concept. 

H2: OPA with TQM has a positive effect on students’ design skills. 

H3: OPA with TQM has a positive effect on students’ attitude toward PBL. 

H3-1: OPA with TQM has positive effect on students’ Inquiry learning attitude. 

H3-2: OPA with TQM has positive effect on students’ Reflective learning attitude. 

H3-3: OPA with TQM has positive effect on students’ Teamwork attitude. 

H3-4: OPA with TQM has positive effect on students’ Creative problem solving attitude. 
H3-5: OPA with TQM has positive effect on students’ Open-endedness attitude. 

H3-6: OPA with TQM has positive effect on students’ Authenticity attitude. 

Update, there is a dearth of studies dealing with the issue how OPA with TQM support worked in collabora- 
tive learning environments of PBL courses and students’ perception about its effectiveness in PBL. It is hoped 
that the current study might unreal and fill this void. It is also expected that the result of this work might provide 
important reference and practical guidance for teachers. 


Methodology of Research 
Participants and the Experimental Setting 


To examine the research hypotheses, a field experiment, a nonequivalent pre/post-test control/comparison 
group design, was conducted in an experimental physics course in the Physics program at a university in Taiwan. 
Figure 1 illustrated the experimental design. The independent variable focuses on two different approaches, 
“OPA with TQM” (with TQM support) and “OPA without TQM” (without TQM support). 

63 undergraduate students in their junior years who were enrolled in the course were recruited in this 
study. Participants were assigned to the experiential and control group based on their own willingness and 
choice; 30 students voluntarily took part in the experimental group (with TQM treatment), while the rest of 33 
joined control group (without TQM supported). Students were grouped into 14 project teams, with team size 
of four-to-five students for the pedagogical advantages of small-group learning (Springer, Stanne, & Donovan, 
1999); For the purpose of experimentally evaluating the actual effects of OPA with TQM approach, a PA with TQM 
support activity was designed in this class. All 30 students in the experimental group completed all the steps in 
the procedure, and the experiment sessions were implemented for 8 weeks for each class. 
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In the study, a simulation software, APSYS, which was developed by the Crosslight Software, Inc. (2006), 
was adopted to develop students’ professional skills in OLED design. Thanks to the merit of economy in time 
and cost saving, APSYS had served as an excellent candidate for to the implementation of PA. 

The production of optical semiconductor devices requires repeated iterations of the steps in the design, 
of crystal growth and test to achieve the desired performance. This process could be very time-consuming and 
expensive. If the optimized design can be achieved by using simulation software before mass production, the 
time and cost required for the development of the devices can be markedly reduced by the elimination of un- 
necessary trial fabrications. By applying simulation software, the Industry—University Collaboration model has 
been successfully developed and implemented (Chang et al., 2011). 


Performance of project - based Learning (Independent 


Assisted Learning approach (dependent variable) variables: 
*Experiment al group: apply “ OPA with TQM” in 


PBL 1. Students’ design concept 


2. Students’ design skill 


*Control group: apply “OPA only” in PBL 


3. Students’ attitude toward PBL (Inquiry learning, 
Reflective learning, Teamwork, Creative problem 





Figure 1: Research architecture. 


Procedure and Treatment 
Pretest for experimental and control groups 


In the first two weeks, students in both the control and the experimental groups had same teacher and 
procedure. First, in the course orientation session, the professor explained the objectives and went through 
the syllabus of the proposed course as well as the requirement in completing the OLED design project. How 
to implement OPA and how to operate the simulation software were also explicated by the professor. The OPA 
procedure for both groups which included six steps were all elucidated to the students: (1) teacher introduced 
the assessment criteria of the OLED design project; (2) students upload the completed group assignments to the 
online assessment system; (3) professor assigned 3 peer group evaluators for each project; (4) group evaluators 
give grades and provide comments on the projects; (5) professor inform each groups of their grades and com- 
ments; (6) professor asked groups to make corrections or modifications based on the comments and group 
upload the modified assignment for an additional round of peer assessment. 

Following the course orientation session, the professor introduced and explained basic OLED design 
knowledge including: basic structure of an OLED and its operating principles, OLED materials, the emission 
spectrum, the structure of quantum wells, internal quantum efficiency, spontaneous emission, and factors af- 
fecting OLED performance. A scaffolding example of how to improve OLED performance was demonstrated to 
both groups. 

After the two-week sessions, to explore whether or not these two groups had similar antecedent knowledge 
level, a pretest exam of OLED design concept--were administered. 


Experimental group 


The instruction procedure for the experimental group included four steps. 
Step 1. Collaborative skills training: 


In order to enhance students ’skills in collaboration team working, the experimental groups was instructed 
skills in Communication, leadership and conflict management using the Deming Cycle instruction materi- 
als. The training material was versatile which integrated films about the Quality Control Circle and role 
plays, during the early stages of learning sessions. 
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Step 2. Benchmark learning 


In this step, a scaffold example of the Deming Cycle was demonstrated to students. The purpose of this 
intervention was to help students effectively modify their project to achieve better performance by 
comparing and assessing other group projects; then students can understand the difference between 
their work performance and better practices of others’ 


Step 3. Checklist: 


Whenever a new design was developed by others, the students were asked to discuss why the perfor- 
mance and parameters were different between their projects and the work of others’ This comparison 
was made through using of a scoring rubrics developed by Chang (2006). This rubrics stimulated students’ 
reflective thinking upon what the differences were between these OLED designs and gave lights to which 
parameters needed to be adjusted to optimize OLED performance. 


Step 4. Continuous improvement: 


In each assessment, PDSA sequence was repeated at least twice. Students in the experimental groups 
were asked to discuss and review quantitative data and qualitative recommendations from other groups 
to identify relevant questions establish problem-solving strategies, and set goals for the next step. 


Control group 


Students in both the control group and the experimental groups had same curriculum, teacher, having taken 
the same courses in the project course. For the development of the projects, the project teams in the control group 
followed the ordinary OPA procedures as mentioned in the pretest step. 

The project teams in the control group were also expected to work together to carry out the project, though 
they were not given any particular interventions of instructions to follow in this collaborative team projects. When 
students encountered problems, they sought out each other's help since collaboration was a requisite of this course. 
.In this situation, collaboration was transient and not formalized in the project teams. 

The only difference between these two groups that students in the control group, were assessed only by 
“Final Scores from Teacher’, unlike their experimental counterparts who undertook the “OPA with TQM” approach 
and the aforementioned scoring formula. 


Post-test for experimental and control groups 


Finally, students in both the control and experimental groups were given a post-test, to appraise their knowl- 
edge, skill, and attitude, after a three-round OPA. 


Measurement and Data Analysis 


Table 1 listed the study evaluation instruments in this experimental instruction, including knowledge maps, 
a photonics scoreboard, and the CPLES. 


Table 1. Description of research measurement. 


Objective of evaluation Evaluation instruments Description of the instruments 
Knowledge OLED design concept test Examine students understanding in OLED design 
concepts after the completion of projects 
Skills eheiGnicsScorebeare Evaluate the OLED design skills by expert rating 


SCOres 


Constructivist Project-based Learning Environ- Examine students’ attitudes toward the project- 


PAIGE ment Survey (CPLES) and in-depth interview based learning environment developed by this study 
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To evaluate the effectiveness of the OPA with TQM approach in PBL, students’ OLED design concepts, OLED 
design skills, and learning attitude were evaluated respectively by the OLED design concept test, a photonics 
scoreboard (Chang, 2006), and CPLES (Chang et al., 2011). The pretest and post-test in OLED design concept tests 
were applied to help evaluating students’ understanding of the OLED design principles. The photonics scoreboard, 
built on an Internet platform, was adopted for peer and expert/teacher assessment. The CPLES was developed 
to investigate students’ attitude toward the PBL environment (Chang, 2006; Chang et al., 2011). Students were 
asked to respond on a 5-point Likert-style scale (from 1 to 5) with 1 indicating “Strongly disagree” and 5 indicating 
“Strongly agree”. Reliability analyses and exploratory factor analysis of the CPLES revealed that the Cronbach a for 
the whole instrument is over 0.95, and the amount of variance explained by factors was over 62% for each field 
test. these adequate psychometric properties demonstrated that CPLES was an appropriate measure for assess 
students’ attitude. 

To examine the effectiveness of the experiment, a series of t-tests was carried out to identify between and 
within group difference in the dependent variables. Results of the statistical analyses were illustrated in the 
following session. 


Results of Research 
Effect of OPA with TQM Approach on Students’ Design Concept 
Examining whether or not there was any between-group discrepancy in students’ prior knowledge, an 
independent sample t-test was performed. As shown in Table 2, no significant difference was found between the 
experiment and control groups in their scores of OLED design concept (t=-1.225, p>0.05), implying that students 


in the two PBL groups had similar prior knowledge levels in OLED design concept. 


Table 2. Result of t- test of the OLED design concept, comparing pretest for the two groups. 


Group N Mean SD df t value 
OPA with TQM 30 47.60 12.891 
61 -1.225 
Only OPA 33 O1.2/ 10.884 


A second independent sample t-test was conducted to examine the effectiveness of the experiment using 
participant's OLED design concept post-test scores as dependent variables. As demonstrated in Table 3, post-test 
scores of the two groups were not significantly different (t=-1.595, p>0.05). The results depicted that the OPA with 
TQM approach had no significant effect on students’ OLED design concept. 


Table 3. Result of t test in students’ OLED design concept-post-test between group comparison. 


Group N Mean SD df t value 
OPA with TQM 30 48.53 13.746 
61 -1.595 
Only OPA 33 53.18 9.129 


Influence of TQM on Students’ OLED Design Skills 
The expert rating scores (teacher's assessment scores) on the scoreboard for OLED projects were used to 


evaluate the effect on students’ OLED design skills. Table 4 showed that these two groups had similar design skill 
levels (t=0.768, p>0.05). 
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Table 4. Results of t test in the 1* round scores of OLED design skills comparison of between-group post-test 


scores difference. 
Groups N Mean SD df t value 
OPA with TQM 7 21.14 2.911 
12 0.768 
Only OPA ri 20.28 488 


The 3 t-test was carried out to test if students’ OLED design skills differed in the 3" round of teacher's evaluation. 
Results, as illustrated in Table 5, revealed that there was significant difference in students’ 3" round post-test scores 
(t=3.10**, p<0.01). The results confirmed that the OPA with TQM group significantly outperformed in their OLED 
design skills based on the instructor’s ratings. 


Table 5. Results of t test in the 1* round scores of OLED design skills comparison of between-group post-test 


scores difference. 
Groups N Mean SD df t value 
OPA with TQM 7 47.57 5.740 
12 3,10" 
PBL with only OPA 7 38.14 5.640 


Effect of OPA with TQM Approach on Students’ Attitude toward the PBL Course 


Results of comparison in the average scores as shown in Table 6 indicated that the average scores of two 
groups were between 3.51 and 4.12 (There was one invalid questionnaire in each group), i.e., most students ex- 
pressed satisfaction about PBL course. Moreover, students in the experiment group had significant positive attitude 
toward PBL then their counterparts in the control group in the following dimensions: teamwork (t=5.1, p<0.05), 
and creative problem solving (t=3.04, p<0.05). 


Table6. __ t-test for students’ attitude of the two groups toward the PBL course. 


Subscales Groups 
OPA with TQM OPA without TQM 
t-test 
N=29 N=32 
M S.D M S.D t p 
Inquiry learning 3.66 0.48 3.51 0.56 11 ZT 
Reflective learning 3.82 0.40 3.69 0.47 1.19 24 
Teamwork 4.12 0.38 3.61 0.39 SM .00* 
Creative problem solving 3.93 0.45 oor 0.48 3.04 .00* 
Open-endedness 3.64 0.75 3.64 0.55 01 .98 
Authenticity g.02 0.62 3.01 0.59 07 94 
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Discussion 


This study proposed an innovative approach, which incorporated OPA with TQM as a macro-level instruc- 
tional method. The objectives of this research were to provide instructors who engaged in collaborative project 
learning with new alternative of guidance while they direct their students. It is also hoped that this study will give 
light to students’ continuous improvement and the performance of Project-Based Learning in STEM courses can 
be elevate. 

The results were presented as the following: (1) there was no significant difference in the leaning achievement 
in OLED design concept between the experimental and control groups. (2) for OLED design skills, the OPA with 
TQM group significantly outperformed the OPA-only group. (3) Compared with the OPA-only group, the teamwork 
and creative problem solving skills of the OPA-with-TQM group were significantly enhanced. 

The results of the study provided valuable information and practical suggestions to PBL instructors; it also 
advanced the knowledge of group dynamics of PBL group project that helped to facilitate positive and productive 
teamwork, optimizing the effectiveness of PBL and student's learning outcome. However, there are two interesting 
findings that should be address further. 

First, the proposed approach successfully enhanced university students’ OLED design skills, yet this approach 
failed to achieve its goal in fostering students’ design concept. Examining the mean scores of both the experimen- 
tal and control groups, it is apparent that, though not significant, the control group even had higher scores both 
in the pre and post-test in OLED concept. Why did such a low score group (the experimental group) eventually 
performed significantly better in OLED design skills than their high-score counterparts? Qualitative information 
collected form interviewing the experimental group students provided some reasons for this question: “S7-7 We 
made “smart guess” after assessing others’ work via OPA""S 1-2 we deliberated why and how the other group designed 
that particular project and tried to imitate their viewpoints” and“S 1-5 we adopted the trial-and-error strategy, sometimes 
it worked well.” Learning by trying or doing appeared to be the reason why the experimental group outperformed 
the control group and the reasons might be that students in experimental group practiced their skills by knowing 
how to do instead of knowing why to do. Therefore, encouraging team-—members to be responsible and excellent 
evaluators is of essential importance. Thus, it is suggested that the original scoring formula should be added a 
third component in sources of evaluation; the reward interdependence between assessor and assessees should 
be added to the proposed formula. The formula can be modified as follows. 


Total Score = (Final Scores from Teacher)x(W1) + (Average Progressive Scores of team from three-rounds of OPA)x(W2) + 
(Average Progressive Scores of other team which received our assessment comments from three-rounds of OPA)x(W3) 


Second, the proposed approach had relatively little and non-significant effect on inspiring students’ attitude 
of inquiry learning and reflective learning. The course teacher pointed out that students would not spend much 
time to study this course further since this subject was not a required course for the entrance examination of gradu- 
ate school. Students often concentrate on subjects relevant to the preparation of the entrance examination, their 
next stage life career development. Slavin (1990) and Sharan (1990) also found that a conventional exam-oriented 
education system produces negative effects on learning among junior undergraduate students. 

These findings of the current study are slightly different from previous studies (Chizmar, 1994; Liu et al., 
2001); however, there was no study exploring how OPA with TOM affected PBL in a STEM course. Chizmar (1994) 
showed that teaching strategy integrating TQM could enhance students’ active learning attitude. Liu et al. (2001) 
pointed out that PA is helpful for undergraduate students, and students all show a positive attitude in regard to 
the proposed STEM course. 


Conclusions and Limitations 


An innovative PBL instructional approach, the OPA with TQM support approach, was proposed in the current 
study which provided an empirical investigation on its effectiveness on students’ performance in an undergradu- 
ate STEM course. The particular design of this pedagogy is to tackle down some dilemmas that had been the 
downsides of PBL proven by previous studies such as insufficient communication, uneven distribution of tasks, and 
the free-rider problems so that the efficacy and quality of teamwork might be ensured. The results demonstrated 
that the team members after implementing the OPA with TQM approach tended to have better OLED design skill 
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performance, efficient team-working and creative problem solving attitude. Thus, the proposed approach suc- 
ceeded in facilitating team members to collaborate for seeking continuous improvement. However, this innovative 
PBL instructional method had no significant effect on enhancing students’ design concept, inquiry learning and 
reflective learning attitude. 

Data from in-depth interviews, serving as complement, indicated that for undergraduate junior students, OLED 
design was a brand new domain and it was difficult for them to finish the complicated projects in a short period 
of time. Some students might have used “smart guess” after assessing other team’s work via OPA. Influence by the 
Confucian culture, Asian students respect their teachers as authority figures and usually over rely on books and 
teachers, whereas they had insufficient initiative and reflective thinking incentives. Therefore, elevating students’ 
learning motivation for PBL regardless of pragmatic consideration of whether such subject is required for graduate 
school entrance exam for their pursuit of higher degrees is an important issue for higher education in Taiwan. 

Although the current research adopted a rigorous research procedure, some limitations must be taken into 
consideration. Extrapolation of the results must be approached with cautions. Future studies are suggested to ad- 
dress the following issues. First, the findings and implications are obtained from merely one particular computer 
simulation software (i.e., APSYS); representativeness of the studied sample is limited since the sample size was 
not big. The implemented experimental instruction was a quasi experiment design, not based on strictly controls 
which paired up equivalent characteristics of individuals for comparison; Individual difference variables, such as 
personality, learning style, as well as group effect, such as group decision-making or team processes, could be 
incorporated in future studies. Second, the training course and procedure of TQM implementation could be stan- 
dardized and computerized to progressively refine and sharpen the focus of peer judgments through discussion 
and reflection. Third, the students in Taiwan have unique value and behavioral patterns, such as they had diverse 
definition of learning achievement and diverse career pursuit. to be more inclusive of the target population that 
representative all walks undergraduate PBL and to be more confident of the research generalizability, cross-cultural 
validation studies using large sample is highly recommended. 
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